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AB.STPAC^i 

The purposes of the Workshop on Measuring the Quality 
ot Public library i:ervices were to explore attitudes toward service, 
discuss the many facets of measuring library services and get a 
close-up view of one particular method of evaluating services: 
"hidden testing." Informal proceedings ot the workshop include: (1) a 
talk of Ernie DeProspo entitled "The Measurement Art," giving tije 
general state of public library measurement and some new techniques 
that are currently being explored, (2) twc talks by Tom Childers, one 
entitled "Measuring Quality of Information Services," discussing 
recent s-^udies that have involvea the hiaden testing of library 
services, and the otner entitled "Measuring Impact and •Quality* in 
Maryland," summing up the workshop and offering recommendations for 
future activity, and (3) "A Very Ghort Bibliography on Library 
Measurement with an ^mphasis on bidden Measurement." (Author/KE) 
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FOREWORD 



In these pages you will find what is most aptly called 
a Report of Unproceedings of the Workshop on Measuring 
tlie (Quality of Public Library Services , (Nev; Carrollt('n, 
Md., June 1^-15, 1972), The "introduction" and Ernie 
DeProspo's talk have been reconstructed pretty much as 
they were heard, "Measuring the Quality of Library 
Services" is altered. And the "Closing" won't be recog- 
nizable at all^fcr it was never delivered, we decided 
to take advantage of having the group together in the 
Workshop, and to discuss rather tlian lecture. The 
"Bibliography" you find here is larger than the one you 
were sent before the Workshop, we hope that it will carry 
you into new explorations. 

So these Unproceedings are partly a report and partly an 
extension of the Workshop — intended to preserve some of 
the spirit of inquiry that developed there and to provide 
some guidelines for going beyond the limits of the Workshop. 




PRCXJRAM 



Wednesday, June 14^ 1972 

1. Prologue and Introduction. (H. Thomas Walker, Division of 
' Library Development and Services, and Tom Childers) 

2. The Measurement Art: The general state of public library 
measurement and some new techniques that are currently being 
explored. (Talk by Ernie DeProspo; large group discussion) 

3. Measuring the Quality of Infonriation Services: Recent studies 
that have involved the "hidden testing*' of library services. 
(Talk by Tom Childers; large group discussion) 

4. Discussion of hidden testing: purpose, general methods, limi- 
tations, etc, (Small-group discussion, led by Ernie DeProspo 
and Bill Summers) 

5. Briefing before telephone testing. (Small-group discussion, 
led by Bill Summers and Tom Childers) 

6. Testing, pt. I: Simple requests for "facts"; Debriefing. 

Thursday, June 15, 1972 

7. Briefing before Testing, pt. II. 

8. Testing, pt. II; Debriefing. 

9. Participants* conclusions about hidden testing: uses, limi- 
tations, practicality, etc. 

10. Measurement and the Quality of Maryland's Public Library 
Service. (Semi-formal presentation by Tom Childers and Bill 
Summers) 

11. Sign-off. 
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INTRODUCTION 



by 

Thomas Childers 



For a long time, I never ran across anything that shook my confidence in myself 
as a librarian, I was taught in library school— and I guess I really didn't 
want to challenge that seriously-- that a "professional" librarian was better in 
one way or another than a "non-professional" librarian. I was taught that if I 
performed certain so-called professional tasks when I got out into the field, 
that I would automatically serve the public well. Oh, there were accepted ways 
to question what was going on. We were taught to be dissatisfied with the num- 
ber of people we were serving, and we were expected to be very critical of 
Library of Congress classification, I don't think anyone ever mentioned to me 
the idea of periodically testirtg myself or my employees on what they were doing. 

The first thing that made me uneasy with Me-as-Librarian was a report by Charles 
Bunge, He had tested some professional librarians and some non-professional 
librarians in a pretty well controlled study. He was trying to find out if the 
professionals were any faster than the non-professionals, and if they were any 
more accurate. I am not sure why I was reading the thing in the first place. 
But I remember my reaction to the conclusion: librarians are no more accurate in 
providing reference service than non-professionals; they are a little fast<»r. 

This shook me. It was probably the beginning of my skepticis;n. 

Since then, I have done some research of my own which has upset my complacency 
even more; I will tell you about that in more detail later. The Bunge study 
and some few others, including my own, have begun to change my attitudes toward 
what service is and what service should be. One purpose of the Workshop is to 
explore our attitudes toward service. 

Of course, since my own attitude would probably not have changed much without 
that Bunge study and without some subsequent other unsettling studies, I feel 
very strongly the need for systematic measurement in libraries. The purpose of 
this Workshop is to get us to sit down together to talk about our needs for 
measurement-- if there are any-- and some of the ways that we might go about 
measuring services. It behooves us to do so at this time, not only for our own 
professional curiosity and our professional commitment towards quality service 
but also to respond to one of the growing trends in management: PPBS. Its 
growth (and I assuming here that this growth will continue) necessitates the 
development of quantitative measures that go beyond merely counting transactions 
in our libraries, and begin to say something about their impact . 

Another purpose of the Workshop is a very personal one. I want to communicate 
with you. And I want you to communicate with me. T»iere is considerable Iso* 
lation of formal library education from library practice. At this point In my 
own evolution, I'm not at all interested in one controlling the other. I am 
interested in sharing our knowledge and experiences toward constructive ends. 
I don't think any of us needs another academic/practice confrontation filled 
with the ordinary polemics and diatribes. 



I have found (surprisel) that the academics do^ have Svvc things to con rrbute. 
I have sort of pretested this or part of this Workshop in a cc!'^)le ot j laies 
where large and serai-large groups of honest-to-God practicinr. libi-ariars -joro 
assembled. I have remarked at the incredu' js reception of sone of the i ^- 
search findings. The librarians were outraged, enbarrassed , non-plussec! , et 
al. To an academic, it's sort of nice to have a strong '^caction— even a 
violent negative or.o. All these .<;oals and fceJinc:s led u\t^ the doorsteps of 
the state of Maryland. And to make a long story short, tht State boup:ht the 
idea and LSCA is paying for it, and Drexel — that's ne-- is doinj> it. And, 
after approxLniately one year of piannin'^ (incredible?) here ^'e are. 

You're going to try to do at least t\;o different things in cur two days here. 
First, we want to talk about the ^^eneral business of measuring library services. 
How do we measure our services now? Why? Are th'^.re better ways of measuring 
services? l^at looms on the horizon? 

Second— and this is going to be the action part of the Workshop— we'll get our 
hands on a particular method of evaluation that has been tried in the last few 
years. In this second "action" part, it's important that we know what the 
specific goal is: that is, we are not hete to evaluate library services in this 
or any other area. We are not here to draw conclusions about the level of 1 i« 
brary services. We are not here to make it hot for the folks in the State. But 
we are here to talk with each other about measurements in general, and about a 
specific kind of measurement. We are here to explore that specific method, to 
see how far we can push it, to find out how much we can learn from it, to find 
out what it can do for us as librarians in making us better on the job, to find 
out what kind of management information could be gotten from this kind of 
evaluation. 
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TlIE MEASUREMET'IT ART 
by 

Ernest DeProspo 



In the public sector generally, and the field of llbrarlanship specifically, 
the term 'Sieasurement" tends to generate a variety of emotional images-- for 
the most part negative ones. For some the tern is vieved in a narrow context 
with connotations of highly sophisticated mathematical formulas. For others 
its meaning is so Imprecise as to be useless* The ooint is, however, that for 
far toe many librarians measurement is t)rpically seen as a threat to both their 
personal and professional endeavors. The question of measurement, then, must 
first of all be discussed from this emotional view before we can proceed to any 
rational discussion of its merits and limitations. 

In a teaching situation it is often popular in dealing with any particular sub- 
ject to ask whether or not the subject is "an art or a science." In the area 
of measuring performance of library activities the response is a simple one: 
It certainly isn't a science, and we are not sure to what extent it even ap- 
proaches an art. Part of the dilemma for the librarian is the historical 
fact that from the 19th Century on, we have assumed a habit towards measure- 
ment in evaluating library programs. This habit exists in the form of presenting 
simple statistics on either the local, state, regional or national level which 
imply a direct connection to performance. And part of the dilemna we face in 
overcoming some of the emotional issues inherent in measuring performance re- 
lates directly to the historical precedent of having established a procedure 
(collecting and reporting statistics) which we have come to reject as a mean- 
ingless exercise. 

There are three main elements to consider within the question of measurement. 
First and most important, measurement assumes "evaluation." In effect, it 
would be inconsistent to speak of '"measuring" this or that service, or this or 
that program, or this or that performance, and not to assume that an "evalua- 
tion" is being made of this or that service, program or performance. Second, 
"evaluation" assumes the desire to know where we are at a particular point in 
time. And thi.-d, there is the recognition that som.e change is needed and 
wanted. We should point out, however, that a key issue is that the recognition 
of and desire for change need not arise from the in-group. The need for change 
or demands for change need not come from the librarian. I think you will agree, 
however, that it is obviously preferable if the recognition and the initiation 
of change comes from the profession itself. The point is that if we can over- 
come or at least better understand some of the emotional issues, we can pro- 
ceed on a course which would clearly be within our own interests. 

Measurement, then, basically deals with these three question, where are we, 
where should we be, and what are the kinds of things needed to get where we 
should be. Improvement in performance^ improvement m programs, improvement 
in services, assume that we can confidently estimate levels of performance at 
a given point in time. Then the proiesslonal judf^ement may be made: it is 
cither good enough, or it is not good enough and certain steps will be required 
to change it. Implicit in this intellectual approach is the realization that 



every change --ust dtjoll -/ithin the realn of scarcitv and that there are to ho 
losses and gains as a result of the process. 

What is involved is really asking the essential question: \^.at the devil is 
going on here? Tlie question suf^gests that there arc limitations to our ccrr-on 
sense, that a distinction between what we know and what we think we know must 
be niade. There is verbal evidence which suggests that there are some serious 
v;eaknesses between how the individual providing the service thinks ttiings are 
going and what in fact is going on. One need not go over the extensive exan- 
pies of the considerable gap between, for exainple, the librarian's viev; of who 
is using the service and who, in fact, is using the service. Again the point 
is that cortnon sense will only take us s^o far in answering the question of what 
is going on here, where we should be, and what changes are needed. 

Typically, wher. we discuss the measureDeni of library services, we can antici- 
pate a set response. For example, in the classroom the student's initial re- 
action is often one of bewildermenc: Why do I have to study this stuff? I 
just want to be a librarlani That response typifies the concept of librarian- 
ship in its most narrow sense, bound by the walls of the library itself. 
Simply stated, it is an attitude that is inward-oriented. It tends to rein- 
force the observation made by Pierce Butler in 1933 when he characterized that 
kind of librarian as an individual happy in the "simplicity of his pragmatism." 
The point is that — particularly today and for the foreseeable future— "sim- 
plicity of one's pragmatism" is equivalent to a Utopian view. That is to say. 
It is fundairentally non-pragmatic. Within the context of this workshop, the 
fact that you are here suggests chat you haven't foreclosed on the possibility 
of looking at v/hat you have been doing somewhat differently; that you are open 
to some suggestions for evaluating services (In this case, reference service) 
perhaps differently from the way In which you have been evaluating them; that 
the issue of measurement remains open; and that fundamentally you are interested 
In looking at this whole Issue of evaluation as a way of improving the perfor- 
mance of libraries as they attempt to deal with the multiple demands and needs 
of Its clientele. It Is within this context that I think we need to look at 
some of the effective barriers to evaluation and to pull these out in the open. 

Effective Barriers to Evaluation 

There are four or five barriers that warrant particular Identification. Jerry 
Walker, In a paper entitled "What's Wrong (Right) with Evaluation" (In Anna M. 
Gorman, editor. Proceedings of the National Research Conference on Consumer 
and Homemaklng Education . Columbus, Ohio: The Center for Vocational and Tech- 
nical Education, Ohio State University, 1970.) suggests a variety of issues 
which Influence* the process of evaluation, including some of the basic barriers 
to effective evaluation. Although placed in a different setting, I believe 
that a n'jmber of points raised by VJalker arc appropriate frr our consideration 
of measurement. 

THE FIRST and the most fundamental barrier Is that evaluation Is a threat. 
Within llbrarlanshlp the feeling of a threat Is exacerbate^ I^ecause we tend to 
"personalize" the things we do. That Is, we tend to be subjectively oriented 
and, as a consequeiice, are afraid of criticism. There is a general negativism 
that underlies critiques of what we are doing in our professional activity, and 
I think It is exacerbated because we have not developed (from an educational 
point of view) a mdre objective base from which to look at the proc^3S, 



If evaluation is to be considered S'»riously, it must necessarily enbody the 
concept of accountability. Evaluation is a way of assessing the extent to 
which any program, project, activity, service, etc, neets its own specifica- 
tions, its own objectives. And in ordei to see if these specifications are 
being met, they first must be stated in clear and demonstrable terms. For ex- 
ample, within i:he area of reference we would say that a demonstrable objective 
is to satisf> 75% of all the reference requests; in circulation, we could say 
that the demonstrable objective is to satisfy 807a of all circulation demands; 
and so forth. 

Historically, public agencies (and libraries are no exception) have been very 
hesitant to state their objectives in terms as precise as those. The fact is 
that we don't really like to set forth our intentions in measurable terms. 
And, until recently at least, it has probably been functional not to do so. 
Today, with the increasing strains on fiscal resources, with increasing pres- 
sures to open up the decision-making process as libraries are forced to relate 
more to other institutions, we nay see some significant changes. For example, 
the pressure to move into PPBS, and the very poor experience which we have had 
with it so far, relates in part to this issue. That is, we really don't have a 
data base upon which to comply with the PBB requirements. We don't have the 
experience in generating the data needed for matching against the objectives 
of that kind of budget. 

In other words, measurement and evaluation demand that we do just what we have 
historically avoided doing. And, vhcn you couple this situation with the c- 
motional issues involved in measurement and evaluation, you can see that change, 
if it is going to occur, will probably have to take place in stages. Ue will 
need to develop appropriate strategies and tactics to decide which areas, which 
activities have top priority for gaining the kind of experience which will 
allow us to set forth procedures in terms that we can then translate into 
measurement indicators. Again, the point is that we need to know xAiere we are 
first before we can proceed to the next step on a logical basis. Typically, 
needing to know where we are first is often the less interesting question. We 
would much rather know the meaning of this or that, and there is tremendous 
pressure to skip steps 1, 2, and 3 and jump into 4, 5, and 6, I firmly believe 
that skipping these basic steps has, in the long run, been self-defeating. And 
let me no tP that we probably can be much less ambitious than is commonly assumed 
in deciding what kind of basic information is needed before proceeding to some 
of the more meaty issues that face us. 

It is important to point out once more that both as individuals as v^ell as pro- 
fessionals, very few of us want to be evaluated. Teachers don't believe that 
they real'.y cau be evaluated effectively as teachers; students reject the 
notion that gracing is fair or appropriate. We have tc face openly and honestly 
the reality that evaluation is considered a threat if xje are going to overcome 
one of these effective barriers. Unless \je do so, any rational attempt to 
evaluate what we are doing in libraries is likely to fail, 

A SECOND obstacle to evaluation is that the library decision maker or policy 
maker can respond to situations much better than he or she can plan and create 
a situation, llie routine pattern of any activity (again librarleiS being no ex- 
ception) tends to create actions which are really programmed. The problem with 
change is that it deviates from .he norm. It tends to heip:hten the insecurity 



and the doubt because uc can't resort to a prof:rarv>e<' response. !vC ha-e . - 
up with a non-pro^;rarB^ed response. A- J this is alvays -^oro 'iiTirult, r- 
certain, rore t'.reatenin^;. V*c tvpica.'l cannot specif - iniornatinn m'Ods c : - 
plecoly ar", accurately in advance, or specify the t\'pes, the priorities, ar ' J^r 
xreiehts of th.c criteria T/hich we use in nar;inr^ our decisions. Yet on a p^c:;^- 
facto basis, ^^o tell the e^aluatvr what r\ic not ',"int: and what var- nissl'^'^ 

A THIRD barrier is that an evaluatio'-' server- r^ultinlc r>prr'^*^es r.rd these pur- 
poses often contra^rict one another. For exarple, i-lLbii. .iiticular pro^rar. r 
ser^'ice. you could think of a supervisor who i^ohr at the purpose as reallv in: :: 
of superficial, the evaluation as really hcin^, done to appease sone fundin;^* 
Z'^fiXit. The evaliiator nay view the purpose aii a careful r^^^nortin'^ of t' o ii^l'^M- 
nation required to nakc better decisions. A client of the .-service nay vic\y «-he 
purpose as unimportant and just another waste of tax dollars. These confH^.Lin<» 
attitudes are cc^nnon in any evaluation process, and because of them the c^^alua- 
tion tends to f^eneratc far nore questions than iL fettles. 

THE FOURTH harrier to evaluation is nonev; it is an e>n'ersive proposition in 
tern? cf tiir.e, personnel resources, and T?*onev. And in tipe of scarce resources 
evaluation is often seen as a luxury which can be slou'^^he! off wher the belt 
needs tlghteninr^. 

linn;';.. A 7IFTH and L-.portant barrier to evaluarlrn is that il is often seen as 
a one-shot affair. Rr.ther than an on-ftoing operaMi^ -., it happer s once. There- 
lore, the tine, offorr, and emotion involved are net really worth the effort. 
It is not goinf> to hc"^e any continuing influence or effect on the proaran. 

No anount of rhetoric will eliminate the existence of these barriers to evalua- 
tior and neasurenent. The first order of business is their reco'^nitior . Vith 
respect to e^/aluation as a threat, I think we have to accept this reality as 
one of our human frailties. But we also -nust recognize that if evaluation is 
initiated and controlled wisely by the profession, rather than bein,?; conducted 
by some unknown "outsider", it becomes that much less threatenin?;, Aj^ain, I 
think we must realize our need to reduce the subjectivity with which we view 
wnat we do and understand that some objective, systematic evaluation nav very 
well be in our own best interest. 

The observation that the policymakers can better react to situations than 
antic ipate them basically reflects the reality tha*: '':;r >iLn(l-sicht is always 
clearer than our foresight. It is a reality — certainly not unique to li- 
braries— which requires some change in administrative style. It will require 
p«lic:'makers to be more willinn than they have been to p^o out on a limb oc- 
casionally— evan if it cracks. Easier said than done. But vc need somehow 
to fint^ hcLter ways of moving into non-prograrnne<! decisi'^a r^ia^rinn and to 
recognize this effort will require trial and error. 

Th?. ''acts that evaluation serves multiple, conllictinf^ purposes cv^. Is e?tpensive, 
are both inevitable. Again, reco^ition of these factors wi ! ! ir»duce their im- 
pact. It may he that with respect to costs we have bf'on pennv— '-fsc and dollar- 
foolish. Better knowledge of what we are doing, wh^ \^e are doin<: it and how to 
improve v;hat we are doing may be an investment that t:i71, if not produce dollar 
savinrs, at least produce substantial improvements in our prodncts. 



Ivith respect to the claL-n that evaluation is a unique or one-tL'-^c affair, the 
simple response is that it need not be so and that it is Ics?; likely to he 5^0 
the nore the Inc'ividuals in the process are involved. In our case, tho norc 
the lihrarians in a particular library are involved in evaluating what they are 
doing and in interpreting the results for that particv.lar library, the less 
likely is the activity to he a one-time thin:;. 

The Art of TIeasurement within Librarianship 

I indicate' earlier that \7e have a precedent for measurement dating back to at 
least 1887 with the adoption of a policy by the American Library Association to 
collect statistics for '"measuring" what libraries are doln;^. This historical 
precedent has become a substantial liability in moving; to other forms of 
evaluating and measuring performance. The traditional approach — counting 
things and reporting these things in rone kind of table or form — has generated 
a process of looking at what we do from an Inward rather than from an outward 
point of view. Simply stated, we tend to be ''thing" or input-or iented ; we 
assume that these inputs equal service. For example, we assume that the number 
of books that we add (which Is an Input) equals or has something to do with the 
quality and quantity of service (which are outputs); we assume that circulation 
is an indicator of services; we assume that the number of professionals Is an 
indicator of service; and so forth. The point is that nost of our measures 
that we still heavily rely on are not user-oriented. Ke are forced to make the 
fallacious jump from input to output. Yet, as we are challenged by others 
(such as budget agents) to prove our worth, we become less and less confident 
in the relationship between the inputs and r:he services which v/e offer. 

The basic controversy with respect to measurement in the field of llbrarlanship 
can very easily be summarized. The position Is essentially emotional, and It 
Is the most significant argument made. Those who deny that we can measure in 
any quai tltative way what we are doing argue that vou can't infer quality from 
quantity. The problem with that argument Is that, historically, we have been 
Inferring quality from quantity In the kinds of statistics that we report. On 
a logical basis we know that (1) there are relationships between quantity and 
quality and (?) absolute precision Is not only unattainable, but not necessary. 
Our space program tolerates a margin of error, even though this endeavor is 
seen as "absolutely precise." Tlie Issue Is not whether one can infer quality 
from quantity, but what are the professional judgements and interpretations 
required to make what type of Inferences. Our overall lack of experience and 
the growing challenge to the manner in which those in the profession have been 
defendlrg their programs has made us rather timid in saying what. In fact, 
quantitative measures mean. In large part, then, we are dealing In a situation 
in which we simply haven't generated sufficient experience In looking at dif- 
ferent ways of evaluating performance. 

Within our American Library Association *;tudy on the measurement of public li- 
brary ef fectivene«c we have attempted to develop measures which simulate the 
library user as he seeks very basic Tlbrarv services. Our concern is get at 
steps 1,2, and 3 before we nroceed to the more interesting questions that 
are tn step^ 4, 5, and 6. In our extensive literature search on measurement 
we found no public a.'^ency doing well. We are not atypical, then. For example, 
in spite of the enormous dollar outlay, there has been no method devised to 



measure the cf lectiveness of public sc^'o-^ls-- l.-^-'nitc ill the tcstinc; anJ aP 
the claims. Inis fact isn't tc su^-e'^t that '-e sh^M r :.c t-atis-ied h- ' ov 'r- 
we haven't heen (ioinr, but sL-^l^- tr i-idicatc :bat \:e arc ror .^t^T)icaI. 

^'t' finJ that. ncSL oi tuc. ] i'^rar^^ statistics prose^toc^ arc do'^crlptive ar'' aro 
very linited ir the kintis of analysiri \;hic'.: can 1>'^ .-.ajc, Jl.c -cc^ 'or r.orc 
sophisticates statintical approacher is cH Jou':. 

A;ol7inj! a variety statistical test^: to tht- I'.S. OffJce of Ei'ucatiun stat:i^- 
tics, \re ffnd very Jittlc relationship aiion^ the Lter.'S rep'-.rte-J. ihe onb- ncs- 
sihlc conclusion that -'c can arri 'e at is that snc^Urr units of service apnoar 
to be iTiore ef:'ecMvc than Jarp.er unli^'i f^f service. The .stud" has a aunbor ^'f 
constraints — of coi.rse the obvious '>nes are ti: c and -.oney. But -nore Lnp.ir- 
tantly, the decision was lade that the nethods and approaches devoloned he 
Influence J by the fact that the t.ata should be collectable at the local Mnit 
and that final int^^'-^^Titations should be made at the 'ocal unit. It is one 
thiny I'^r a hiyhly sophisticated trained tean < f re*;earchers to rro out and col 
lect inionr.ation, co^ie back, nanipulate the inforniation and cone up with their 
findii.^s. It's quite another thiny lor those who \:eren*t trained In the process 
to do <?o. Yet the fcrrer approach is one which tvpicall> tends to bo a *'one- 
-hot" operation. You seldom see the researcher ac^ain— or more importantly, 
see and imJerstand t! e results oi the research. 

So considerable effort has been place ^ in developing an approach v;hich would 
involve librarians in all phases of the process of gathering the data and then 
in final interpretation of the data. For exaniple, we put on a series on tv;o-day 
workshops in wliich librarf.ans from each of the libraries involved in this study 
was j>ivcn instructions on bow to supervise the collection of information by 
their local library. Rutgers has been active as sort of a central data pro- 
cessing location from wiiich preliminary data analysis will take place. From 
there, information will be sent back to the local library for final interpreta- 
tion by the librarian. 

The s^udy has two distinct phases, broken down into data collection over a two 
v^ek period. In tl:e first week, information is collected on the following kinds 
of items: library sponsored and conmunity oriented prograns; document availabil- 
ity (for document availability we essentially want to know what chance the user 
has in getting a book or a journal article fron that library); the physical 
plant and oqiiipnent; staff, including library experience and education. 

The second week— which involves three days in which we ask each library to 
indicate their high-, medium-, and low-use days— is set up to collect data on 
thr following kind3 of items: how the facilities and equipment are used; what 
the patterns of reference usage are; what outside library circulation lool.s 
like; and what library use looks like. Every user entering the library over 
these three day periods (the high-, medium-, and low-use days) is provided with 
a very small library user "ticket" which gives us some information about who is 
using it-- student, non-student, sex, occupation— and also j>ives us information 
on how long they used the library so that we can say something about that li- 
brary in terr.s .^f time usage. 



We have found the tickets to be particularly discriminating. For example, we 
were able to characterize our four pilot libraries as: (1) a "supermarket" 
operation, in which the average turnover is about 15 to 20 minutes; (2) low- 
turnover, older user group, with the average user in the library close to an 
hour; (3) program-oriented, with highly competitive divisions in the library. 
Each library tends to have unique patterns which might have been used to pre- 
dict such patterns; yet standard statistics or measures did not suggest that 
the patterns existed. 

Overall, we find a very posit tiuude toward the library by the user. We 

see very poor utilization of skill resources (of professionals, in particular) 
in the libraries. In the pilot libraries we find a very low level of reference 
activity. The reference questions, by phone or in person, average something 
like ten an hour. Fifty percent of these questions are directional in nature. 

We are just beginning to analyze the data beyond the pilot libraries, so all 
these comments are highly preliminary in nature. But let me make a iEinal 
point: The high degree of involvement of each librarian has produced rathar 
remarkable enthusiasm. The study reinforced my own biases when we talk about 
measurement and evaluation. If we want evaluation to be an on-going process, 
then it is critical that the librarian be involved and understand what is going 
on. For this reason, I strongly support this kind of Workshop and highly com- 
mend the efforts of the state of Maryland and, specifically, Dr, Childers, 

And a footnote: The matters of change and adaptation are directly related to 
evaluation and measurement. I think we need to take a more than casual look at 
them. Let me conclude with a statement from Alvtn Toffler's Future Shock : 

Our psychologists and politicians alike are puzzled by the seemingly 
irrational resistance to change exhibited by certain individuals and 
groups. The corporation head who wants to reorganize a department, 
the educator who wants to introduce a new teaching method, the mayor 
who wants to achieve peaceful in*:egration of the races in his city- 
all, at one time or another, face this blind resistance. Yet we know 
little about its sources. By the same token, why do some men hunger, 
even rage for change, doing all in their power to create it, while 
others flee from it? I not only found no ready answers to such 
questions, but discovered that we lack even an adequate theory of 
adaptation, without which it is extremely unlikely that we will ever 
find the answers, (Bantam edition, p. 3) 



MEASURING VAE QUALITY OF INFORMATION SER^^ICE 



by 

ThfTmas Childers 



The History 

The measures of librar;' '^service** in the past and today are nothing rnuch more 
than simply counting, count the numbers of documents circulated, hooks 

added, dollars spent ^ hours open, square feet, chairs, and on and on. 

For many years we have assumed that **bigger is better." We have assumed that 
more circulation r^^^ant better library service; that nore books per capita, more 
dollarr per capita, more users per 100,000 population, more hours open — that 
these add up to better library service. It has only been within the last few 
years that there has been even a piddling eftort to test this assumption: Does 
bigger mean better? 

Moving a step further: What does "better" mean, and finally. How good is 
"better?" As we continue to measure the so-called quality of our services in 
terms of simple size and frequency, we seem to reinforce the notion that the 
quality of our services is determined (caused) directly by numbers — tnat service 
is "Improved" by getting people to take out more books, or by registering more 
borrow3rs, or by acquiring more books, or by handling more reference questions^ 

Yet, somehow we know that no one of these remedies-- nor perhaps all of them 
together— necessarily means that we are going to improve the level of the ser- 
vice that we provide to the public. Somehow we know that these factors may not 
be critical in determining the quality of service that we offer. 

Here we sit observing the things that are most observable. Which is natural. 
And it certainly tells us things about demands put on the system, queuing, trans- 
action frequency, carpet wear, etc. But to make statements about the quality of 
our service, I think v^e must begin to go beyond simply tallying this or that 
thing that can be easily tallied. Evaluation of service quality is probably 
going to be more ccmiplex. 

I don't mean to suggest that the library field is ;any worse off than almost any 
other field o It turns out that even some executives of profit-making corpora- 
tions are having difficulty in recent years of defining just what is "profit." 
l/hat nay appear to ])e profit in the short term can very often turn out to bo 
loss in the long term. The gross national product (GMP)— which at one time 
seemed to ha a very respectable measure of the quality of American life-- in the 
last few years has come under considerable attack. 

We may all be in th! s thing together. Maybe net profit, GNP, and the number of 
books circulated in a year are none of them useful in talking about the effect- 
iveness of our corporations, or the quality of ot;r life, oi ^he level of our 
library service. 



Tlie measurement of reference/information services is very much in the ballpark. 
Since the early 30*s, we have tallied the numbers of questions answered and 
unanswered, kinds of sources used (dictionaries, almanacs, indexes, etc.), the 
number of sources used, the kinds of questions answered and unanswered, and so 
forth. 

Measuring the Quality of Information Service 

It seems that we never really asked ourselves how well we are providing cur 
reference and information service. 

Testing the performance of people at the reference desk began— as far as the 
literature goes, anyhow-- in the late SOj^s with the report for the Commissioner 
of the New York Department of Education. In trying to get at the compara- 

tive levels of services from library to library in New York State, some testers 
went from library to library and asked a numba*: of questions. The librarians 
were supposed to answer the questions as quicr<iy as they could within their own 
individual collections . 

The performance of the libraries was not specifically recorded. But what is 
most important about this study, anyhow, is that it was done. Cue small part of 
library services were being tapped at the point where it meets— mi^^ht meet — 
the client. .urrah! There are some problems in the design. of this study that 
appear on the very surface: the testers were identified as testers; and the li- 
brarians were limited to what they could find within their own collection. The 
situation must have felt to the librarians like a Graduate Record Exam — or at 
least not exactly like a bona fide encounter with a real live client. 

Around 1965, Enoch Pratt Free Library tapped their own telephone reference lines 
over a period of four months. They concluded that about one-third of the 
questions were not answered satisfactorily (whatever that means). It was never 
broadcast through the land; probably the land wasn't ready for it at the time. 

Long ago, those cream-colored plastic towers of academia, the social psycholo- 
gist coi.^d the phrase '*co^itive dissonance." In English that meant something 
like "shock" plus "disappointment." Can you L-nagine the amount of cognitive 
disionance— shock plus disappointment-- that would have run through library- 
land if Pratt had broadcast its results? After all, at that tiiue — and for the 
most part, still-- we like to think that we are performing about 1007o, 

(An aside: Last year I did a very tiny study of a city library and some suburban 
branches in the mid-west. The results of the testing that I did were very dis- 
appointing-- that is, shocking — to the central library director. I am certain 
that he would have had my head or reputation or both if I had not mentioned 
Enoch Pratt's study. I was very happy to note the strong effect of such a simple 
statement as 'Veil, when Enoch Pratt studied itself, it seemed to ! e performing 
at a 66% level of success." That irate director got as calm as a kitten; in 
fact, he started to think that his performance wasn't so bad after all.) 



Findings In the Recent Past 

Terry Crowley at Rutgers in 1967 got interested in the idea of acting like a 
client and then judging the kind of response you get frum librarians. So he goc 
a bunch of questions together and bunch of shllls, or proxies, or shoppers, and 
started asking those questions at libraries in Pennsylvania and New Jersey. 
Some of the questions they asked In person and some of thein they asked over the 
telephone. They includec^ all kinds of questions, but they were mostly questions 
whose answers had changed in the last few days or last few weeks or last few 
months. Crowley's Interest was in finding out now well librarians kept up with 
the world as it is. He had a suspicion-- he really had a point that he wanted 
to make-- that when, say. World Almanac Is put into a paper binding, a part of 
every librarian is bound right in there with it and can't escape. A "fact" that 
is anchored in World Almanac remains a "fact" until the next edition, v/hether it 
has changed in the real world or not. To some librarians, a local senator who 
has died is not dead till duly recorded in a library "tool." Or, a change in 
monetary systems in Australia does not "exist" until it Is published. 

Crowley's thesis was that the librarians are not on top of the world that is 
around then and to that extent cannot be effective disseminators of information. 
At the same time, Crowley was Interested in finding out if libraries that are 
richer performed better than ^^orarles that are poor. 

Hold your hat. The results were: 

First: The richer libraries did not perform significantly better or worse tiian 
the poorer libraries. 

And Secondl: The overall performance was 54.27o. A little better than half the 
answers contained the correct answers. Among the problems in answering, Crowley 
found that transposition of numbers and the failure to up-date the information 
were the most common. 

While dramatic, the Crowley study did not have a lot of questions in it, nor was 
the sample of libraries very large. So I figured what he observed could have 
been attributed to chance. Tha^ is, I was not convinced that libraries perform 
that poorly. 

Until I started doing my own studies. 

I was interested in testing the performance of libraries in the way Crowley had; 
but I was most interested in how performance correlated with a lot of other li- 
brary factors-- such as collection size, number of professionals, books, hours 
open, dollars spent per capita, books per capita, circulation, and on and on. 
I won't entertain you with all of the figures-- one single chart had i,081 
4-place statistics of correlation on it, and had to be spread over seven whole 
pages— even though many of the statistical findings were sort of fun and even 
useful. But what is really important for us now and what-- I must conf ess- 
shook me, and still shakes me a little— is tha performance of the libraries. 

My study was quite different from Crowley's. He had a limited bunch of libraries 
he was asking questions from and a very small number of questions— »hich is 
characteristic of an exploratory study. In my research, however, I hired a 



gang of people to ask 26 questions at each of 25 libraries. VJith a few thrown 
out for this reason or that, the total number of encounters with libraries was 
over 600. 

And what was funny (but then again, not so funny) vas that the libraries in my 
study performed at the same level as in the Crowley study. Depending upon how 
you judge ^^correctness," the libraries in my study were successful between 50 
and 607;. of the time. Unlike Crowley's, though, I did not find much in the way 
of incorrect responses. Usually, either the "correct" answer was given, or no 
answer was given or a semi-answer was given. (A semi-answer night be something 
like, 'Veil, I think it might be .") 

Since I did that large study i.i New Jersey, I have done a couple of smaller 
studies in two states. In one state the ->-esults were approximately of the same 
level, 50 - 60%; in the other state performance level was way down, around 20 
- 40%. 

Before you start throwing things at me or your reference staff, let ne hasten 
to say chat there were lots of limitations in these studies. The questions — 
even though they were supposed to look like real library questions — and the 
callers-- even though they were supposed to look like real library patrons — 
we have to assume did not cover the whole range of possible library demands 
and library clients. It is possible that each study had a number of b5,ases 
within it. To that extent, we are on shakey ground if we go around condemning 
public libraries for providing universally low quality service. 

What can we say? Simply, WE MAY NOT F.E DOING AS GOOD A JOB AS WE HAVL ALWAYS 
ASSUMED. 

If a study that occurs inside of those ivied ivory walls (assuming that it gets 
outside of them) makes us a little uneasy about what Vre arc doing, makes us 
question our performance, then the research may have be3n worth it all. This 
particular method of hidden testing has caused some people to worry a little 
over what is happening in the reference departments. It has caused a few 
people to shout at me and get angry at themselves, because the data that it 
produced is so foreign to them. It is not surprising; it's like going through 
a kim' of culture shock. 

The "Hidden" Technique 

This is coming out backwards. I told you the results of some studies; now let 
me go into how we go about them. As I said before, the method really consists 
of asking a bunch of questions at a bunch of libraries in such a way that both 
the people who asked the questions and the questions themselves appear to be 
bona fide. 

TVenty-five libraries were asked each of 26 questions, by the 26 proxies. The 
questions were for the most part fairly simple. Some examples: 

What is the gap !:pecif ication for a 1963 Chevy Corvair? 

Could you identify this piece of poetry for me, please: 
"0 pardon me though bleeding piece of earth..."? 



What was the federal budget expenditure in 1936? 



How many families are there in (this) county? 

What does the 0,J, in 0,J, Simpson stand for? 

What is the book that the Peter Sellers movie, "The Battle of 
the Sexes,** was taken from? 

What were the exact words of John Kennedy when he was 
asked how he became a war hero? 

One of the major problems in hidden testing is to "control" the application of 
the question. This means that, as much as possible, you not only want a bona 
fide question and a bona fide caller, but you also want the application of a 
question to be consistent from library to library. That is, you don't want a 
pleading, cajoling tone used with one library, and a business-like, straight- 
forward tone used with another, for the same question. So to help control 
for this, we have one proxie call the same question to every library In the 
sample. By no means do you get absolute control In this way, but it is prob- 
ably safe to assume that you will get more consistent application of a question 
this way. 

The caller is also given a specified amount of information to include In each 
question. If the question is: "Is the Australian money system a decimal system? 
he gives nothing more and nothing less. If the question is: "How far is Athens, 
Georgia, from here?" .iothing Is left out and nothing Is added. He does not 
omit any information, nor does he Include any extra informatinn. So, when the 
respondent hesitates and says "My goodness, I just don't know,,." the caller 
doesn't say, for Instance, "I think the Australian money system has changed 
the last few years," He says something neutral and information free like, 
"I'd like to know" or " ," 

This may seem artificial. And It is. But this Is the way we tried to control 
the way the libraries were stimulated, in order to be able to compare the per- 
formance of one library to the performance of another. 

Since we were interested mostly in how correct the libraries' answers were, and 
not in what kinds of questions they would accept, or from what kind of patron, 
we developed a list of questions that looked "legitimate", and assembled a core 
of proxies that looked and sounded real, too. None of the proxies (well, we mad 
a mistake on one) sounded like a school student; and If there were a chance that 
a question might be taken as a request for information in working out a cross- 
word puzzle or a homework assignment, we clo*:hed everything In a "personal 
rationale" that made it sound like a "legitimate" adult request (For Instance, 
"My father needs this information in order to fix his car,"). Likewise, in con- 
structing this rationale, we were careful to prepare ourselves so that the re- 
spondents couldn't expect any further information out of us. For example: 
"Gee, I don't know. It's for my boss," 

To keep from biasing the studies against times when professional librarians 
would not be there, ve started at random times during the day and week In our 
calling. And in order to give the libraries the benefit of the doubt, we al- 
lowed them to take as long as they wanted in answering the question, and to 



call back-- or rather to be called back. Calling back nosed sonewhat of a 
liiriitation, We could not allo\; the library to call us back. In almost every 
instance, even vhen the request was for a mere simple fact, they wanted to call 
us back. But sirxe we were calling many libraries long distance, if we had 
allowed then to return our call, we vould have been given away as non-residents. 
So, to avoid suspicion, we usually nade an excuse as to why we would have to call 
them back. ("Gee, the baby's asleep." "I'm calling from work, and really 
shouldn't have incoming calls.'*) 

The proxie writes down as niuch of the whole encounter as he can^ verbatim. 
Some people can do this quite easily; and it's very difficult for others (me, 
for instance). 

Responses are then judged. For judging the responses it seemed best for me to 
have ready before the question w ; as asked a "criterion of correctness," For 
example, you might accept something as simple as "Shakespeare's Caesar " which 
could be predetermined as correct instead of the longer "William Shakespeare's 
Julius Caesar , act III, scene i," This may seem an awfully fine point, but 
immediately when we goc the answers back it was apparent that to be fair from 
library to library, and to be consistent in judging them, we had to have a 
concrete standard such as the "criterion." We found it worthwhile, even so, 
to have several people judge the same responses. 

What can we learn from these efforts? Up to now, the major interest has been in 
looking at the correctness of the answers supplied by the libraries. But there 
are other facets: speed, attitude toward various •t\'p'3 3' of callers, willingness 
or readiness to refer, consistency of policy and service, etc., etc. Look even 
beyond that, and you will possibly see other things that can be learned from 
"unobtruding." 

This means that we have to make some very personal assumptions about what 
"quality" service means: does it mean speed, accuracy, pleasantness, familiarity 
with materials, skill in interpretation, general knowledge,.,? As much as any- 
thing, we are probably testing ourselves, our own concepts of what library ser- 
vice is or isn't and what we can expect from places called libraries. 

Remember that what you uncover in your digs may be ugly. Of course, there are 
people like the woman who knows the spark plug gap specification for a '63 
Chevy Corvair off the top of her head; she turns out to be an airiateur mechanic. 

But then there'll be the respondent i^o, encountering a statistic that reads 
"$8,493, in millions," as the federal budget expenditure for 1936, reads it over 
the phone as "3 million, 4 hundred 93 thousand dollars." 

And the sweet young (judging from her voice) thing who found the page in Bart- 
lett's with the quotation "C pardon me, thou bleeding piece of earth..," and 
correctly identified it as coming from a work of Shakespeare, 



"But what's it in?" 



'^^ell, I dunno. . .er. . ." 

"Is it part of a play or poem or something?" 

"Uh, well, I can't tell, For other quotations it says the play. 
Here's one, ..one from Macbeth, and Julius Caesar. And Twelfth 
Night. An then there's a lot from Ibid." 

"Ibid?" 

"Yeah. I-B-I-D." 
"Well, thank youc" 
"You're welcome." 



FOOTNOTE 

1 

Report of the Cor^Tti.ssioner of Edi^catlon's Committee on Public Library 
Service . Albany: The University of the State of New York, The State Education 
Department, 195 G. 



CLOSING: liFASURIKG EIPACT "QUALITY" DJ llARYTAriD 



by 

Thonas Childers 



Just like most other active states (as opposed to dormant ones), Maryland has 
been concerned with the level of service it proviaes. Just like most other 
states, too, Maryland has been primarily concerned about quality level, or per- 
fonnance, at a point that is removed frorr the point where the user meets the 
library services. 

Take, for instance, the Statistics of Maryland Public Libraries: 1969-70 > 
Fourteen 'Wasures" that say something abojt Maryland ^s libraries are recorded 
here. Only one (count 'em, one) treats of the user and his use of the library. 
The rest say things about Inputs: boolr£, dallars, rcaff, etc. While we have to 
deal with service Inputs, of course, that isn't all there is to knowing wliat 
we're doing or what we are, by a long shot. 

We assume that all kinds of things ultimately have an effect on the kind of 
service we give. This assumption is reflected in some of the studies done for 
the libraries of this state: 

Feasibility of joint service centers for metropolitan libraries, 
by Duchac in 1968.^ 

Study of library aids, or associates, in Mar/land, by Clark and 
Johnson in 1971.^ 

Proposal for a legislative package by the Commission on Public 
Library Laws, 1970.^ 

These efforts and their recommendations-- and I'm not questioning the need for 
them, or their usefulness-- are presumed to have an effect on library service 
or they wouldn't have been undertaken. Ho^^aver, the recommendations are so far 
removed from points of "impact" on thr actual library user that it's hard to 
predict how he rai£;ht beaetlt from them. 

Tliere is a group of studies in the State that gets closer to the user, though. 

Students and the Pratt Library ."^ Talks about frequency of use, 
how use fits into personal daily life, what the library is used 
fcr, and like that. 

Baltimore Reaches Out .^ Deals with reading; media use; relative 
use of libraries as suppliers of materials, co^ripared with other 
sources of materials (which comes close to measuring the impact 
of library service in general). 

Bundy's metro-Maryland study.^ Primarily deals with the habits of 
users. Says little about the success of service. You might infer 
that the libraries perform well for the few people who are very 
heavy users, and that they perform poorly for non-users. This 
doesn't tell us much about performance, though^ 



Warner's study of ILL requests.' Reports bO% success; implies a 
level of service from the user's point of view. 

We cone even closer in the Consad Report: Baltimore Metropolitan Area Work 
ProCTam , 1970.^ The report recommends a systems approach to planning for the 
development of libraries in the area-- which includes 

1. development of concrete goals 

2. creation of measures of success, and the mechanism for 
measurement ^ 

3. ac^ually measuring 'success* 

If specifically recoramends the creation of a Planning Ir.f'^T-mation System. In- 
cluded as part of this System would be (a) regularly collected data and (b) 
intermittently collected data (probably for the figures that are hard to collect 
all the time). The report echoes other sources in saying that PPBS points up 
the need for more and better data. But most important, the data SHOULD BE 
MORE THAN INPUT DATA, more than books, staff, hours, dollars; and it should go 
beyond the very rudimentary output data that we now have— which, statewide, 
for Maryland is essentially "number of books circulated." There is a need for 
more output data, in more detail, \/ith more meaning (which means, probably, 
closer to the point where the client consumes the service). 

I'^xat we dealt with for two days in the Workshop was just one humble possible 
technique for looking at outputs. Y'^u have been introduced to a technique, a 
tool for internal management and possible for statewide development. You may 
or may not have been impressed once again with the importance of looking cri- 
tically at what libraries are all about, from *:he client's point of view. 
Maybe some of the myths of service have been-- if not dispelled-- at least un- 
settled a little bit. 

Don't stop there. Let me recommend that you (a) communicate with your co- 
workers back hone, and with other libraries in the vicinity. Communicate any- 
thing you want, communicate the Workshop. You'll each have your individual 
message, and it will probably relate to the need to look at service outputs. 
Proceed within your libraries, branches, systems, to question in a heretical, 
sacrilegious way the assumption that libraries are performing 100%. 

Proceed carefully, though. Remember the efficiency expert in the movies of the 
forties and fifties? l^y was he alwaj'S cast as a son of a bitch? Because he 
was. Not due to his own fault, but due to the managers who hired him and who 
weren't careful in using the tools of evaluation that he developed for them. 
The managers in those olden days tended to use measurement ON the people in the 
organization, rather than FOR them. Management is not the authoritarian game 
it once was, of course. But measurement can still be abused. That old 
efficiency game (systems analysis, operations research, PPBS) can threaten 
people as well as help them do a better job. Caveat. 

(b) Start developing your own devices for measurement, things that will help 
tell you what you look like-- especially from the user's point of view. The 
method w have called hidden testing is just one small possible tool. 



(c) Consider vor/.in - lorc closol;' \;itn the I'ni^-orsity *tarylancl L:.brary 
Gchot^l and the State Library tcn;ar(! a stato'.;ic'-*^ effort to 

K c::pcrLient \''ith irioac t anu ^':ualit^'' nca^urcs, nncl 

' ollnct rc^^ularly certain L^.nact anu qaaliro - .Lta for 
sLate\/ido consujnpticn- 

rooTi:oTFS 

' r^T !>.::': , rennerli l\ \ Library ^, .e rvicc- Center for Suburban Maryland Countj^- 
S^ /b tens . ::viitiriorc , 190o . 

Clark, Philip 11. and Johnson, IlilJred Y. A Training Progran for Library 
Associates in naryland . New Brunswick: Bureau of Library Information Science 
Research, Gra^biate SL^hool of Library Service, Rutgers Univert- ity , 1971. 

^Comnissior nn tLc Public Library Laws. Libraries for the People of 
I-lar^^Iand . 1C70. 

liar tin, Lo\;ell A. Students and the Pratt Library: Challenp.c and 
Opportunity , Daltinore: Enoch Pratt Free Library, 1963. 

^Martin, Lowell A« Baltimore Reaches Out: Library Service to the Pis * 
r:?d van Lapsed . Baltinore: Enoch Pratt Free Library, 1967. 

^Bundy, ?lary Lee. Metropolitan Public Library Users: A Report of a Survey 
of Adult Library Use in the T.aryland Baltinore-Washinpton Metropolitan ^rca ^ 
Uni;crsity of Maryland , School of Library and Information Services, 196d. 

A/arner, Edward S. Maryland Interlibrary Loan Network Loan Rec^uests ; 
Systei.i and Units Analysis . Baltimore: Maryland State Department of Edu^^ation, 
Divii^ion of Library Development and Scr^'ices, 1969. 



'^CONSAD Research Corporation. Baltimore Metropolitan Area Library Study . 
Pittsburgh: CONSAD Research Corporation, 1970. 
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A VERY SHORT BIBLIOGRAPHY ON LIBRARY MEASUREMENT 
WITH .\N EMPHASIS ON HIDDEN MEASUREMENT 

(The entries are in order from the most general to the most 

specific. The Bunge and the Crowley-Childers items are 

fairly formal reports of dissertation work in the area of 

testing the "quality" of library service.) 

Line, Maurice B. Library Surveys: an Int r oduction to Tneir Use, Planning , 
Proc dur e and Presen t atio n. Hamden , Conn . : Archon , 1969 . 

In 134 pages, a respectable, straight-forward introduction to the 
study of libraries in more or less conventional ways. Includes 
design of the study, collecting the data and analyzing it, and 
reporting the findings, 

Webb, E; jene J. and others. Unobtrusive Measure: Non-reactive Research in 
the Social Sciences ■ Chicago: Rand McNaliy, 1966, (paper) 

As a whole, it*s a sprightly introduction to research methods and 
well worth buying. The section entitled "Intervening Observer: " 
(pp. 155-164) presents hidden or contrived observation as a tech- 
nique of measurement. 

Drott, M. Carl. "Random Sampling: a Tool for Library Research". College 
and Research Librarie s, 30 (March 1969): 119-125. 

Questions about the accuracy of library records, the behavior or 
attitudes of patrons, or the conditions of the books in the col- 
lection can often be answered by a random sampling study. Use of 
this time and money saving technique requires no specidl mathe- 
matical ability or statistical background. The concept of accv^-- 
acy is discussed and a table is provided to simplify the determin- 
ation of an appropriate sample size. A method of selecting a 
sample using random numbers is shown. Three examples illustrate 
the application of the technique to library problems. — abstract 
quoted from the journal. 

Rothstein, Samuel. "The Measurement and Evaluation of Reference Service". 
Librar y Trends , 12 (January 1964) : A56-472 . 
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